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Introduction 


The Literacy Design Collaborative (LDC) was created in 2009 to support schools’ 
implementation of rigorous college and career ready standards for English language arts (ELA). 
Combining evidence-based principles of professional development with a systematic design 
ocess, LDC helps teachers to embed literacy skill develooment throughout their content area 
urricula and promotes teachers’ engagement in formative assessment and continuous 


nprovement practices. UCLA’s National Center for Research on Evaluation, St 


tandards, and 
tudent Testing (CRESST) has served as LDC’s independent evaluator since 2011 and is 
urrently completing a 5-year, nationally funded study of LDC’s implementation and impact. This 
icy brief Summarizes study results from a large, urban district on the West Coast. The brief 
rts with background on LDC and study methodology and then shares study findings, which 
emonstrate LDC’s substantial effects on student learning. (More detailed results from the study 
e available in two CRESST reports, Wang et al., 2019, and Wang et al., 2020.) 
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Background on LDC 


College and career readiness standards (CCRS) raised the bar on what students are expected 
to learn Johnson & Wells, 2017) and consequently demanded changes in existing teacher 
practices to facilitate such learning (Darling-Hammond et al., 2014; Supovitz, 2015). For ELA, 
teachers were asked to shift their practices to support their students in being able to read 
critically, use evidence-based arguments, and communicate clearly orally and in writing. All of 
these changes pose a challenge to teachers who may have never been exposed to these 
practices in the past as either a learner or a teacher (Edgerton & Desimone, 2018; Pak, 
Desimone, & Parsons, 2020). 


a 


LDC was designed to help teachers meet this challenge. It empowers teachers with task 
templates, ongoing training (Desimone, 2009), and other tools to create and implement 
challenging standards-based modules, which can be seamlessly embedded in ongoing 
curriculum. The units culminate in a content-oriented, evidence-based writing assignment that 
draws on unit readings and feature an instructional ladder designed to help students build the 
tent understanding skills they will need for unit Success. 
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LDC thus addresses one of the key shifts demanded by CCRS, that students engage with 
complex, content-rich texts across multiple disciplines (Chadwick, 2015), and its design supports 
a variety of learning outcomes. Research, for example, demonstrates that the integration of 
literacy and disciplinary instruction serves to increase students’ content knowledge, vocabulary, 
reading comprehension, and writing skill (Cervetti et al., 2012; Duesbery et al., 2011; Reisman, 
2012). Further, by encouraging and assisting content area teachers to integrate literacy 
standards into their courses, LDC both deepens students’ content understanding (Carter et al., 
2007, Cervetti et al., 2012; Draper, 2008; Klein & Kirkpatrick, 2010; Monte-Sano & De La Paz, 
2012; Wissinger & De La Paz, 2016) and helps make literacy development a shared responsibility 
among both teachers and administrators, who can assign resources and develop school 

policies to support reform (Shanahan & Shanahan, 2008; Supovitz, 2015 
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LDC uses backward mapping as a key and well-established design principle (Graff, 2011; Wiggins 
& McTighe, 2005). After identifying a high-quality, culminating writing assignment incorporating 
both disciplinary standards and CCRS, teachers chart the specific knowledge and skills that 
students will need for Success on the assignment. The teacher forms an instructional ladder of 
eading and other activities that will help students develop the requisite knowledge and skills by 
eeping in mind the learning goal, the necessary steps to achieve the goal, and the vision of 
high-quality performance while planning and enacting lessons (Ball, 2000). 


The LDC design process serves as the overarching framework of practice for an extended 
school-based, teacher professional development program that incorporates web-based training 
and learning supports.' Through the LDC CoreTools platform and on-site professional learning 
communities, teachers have opportunities to share representations, practice decompositions, 
view approximations of practice, and learn to use specific scaffolds and approaches to 
instruction (Grossman et al., 2009). These activities are all important since research shows that 
standards-based reforms can pose both technical challenges in aligning curriculum and 
adaptive challenges in adjusting teachers’ rigor and practices (Pak, Polikoff, et al., 2020; 
Supovitz, 2015), and one way to confront these challenges involves ongoing professional 
development that is content focused, and provides opportunities for sense-making (Desimone, 
2009; Edgerton & Desimone, 2018; Spillane, 1999). 


‘LDC implementation varies from individual teacher-based to school- or other group-based, but the I3 study reported here 
used a school-based model. 
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As they participate in the LDC professional learning community, use LDC supports, and enact 
LDC-infused instruction, teachers also move through cycles of continuous learning and 
improvement, as depicted in Figure 1. Teachers begin by planning assignments and instruction, 
which they then implement in their classrooms (Bell & Cowie, 2001; Black & Wiliam, 2009; 
Heritage et al., 2009; Ruiz-Primo & Furtak, 2007). The modules offer teachers multiple 
opportunities to assess students’ learning progress and to identify any gaps o 
misunderstandings (Bell & Cowie, 2001; Black & Wiliam, 2009; Hattie & Timperley, 2007; 
Heritage et al., 2009; Ruiz-Primo & Furtak, 2007; Sadler, 1989). Teachers then are encouraged 
to use these data both to revise their modules and to plan subsequent instruction for their 
students, using effective formative assessment practices (Bell & Cowie, 2001; Black & Wiliam, 
2009; Ruiz-Primo & Furtak, 2007; Sadler, 1989). 


FIGURE 1 - LDC Instructional Cycle 
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The LDC implementation model for the I3 study reported here assured that participating 
teachers at each school had dedicated, common planning time; regular Support from LDC 
coaches; and a school-based LDC coordinator. The model also specified LDC’s expectations for 
schools’ and teachers’ participation, as shown in the study’s logic model (see Figure 2). Available 
research concerning job-embedded learning (Croft et al., 2010; Zepeda, 2019), professiona 
learning communities and teacher collaboration (Akiba & Liang, 2016; Darling-Hammond et al., 
2009; Goddard et al., 2007; Lomos et al., 2011; Patrick, 2013; Ronfeldt et al., 2015; Vangrieken et 
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al., 2015; Vescio et al., 2008), teacher coaching (Allen et al., 2011; Cohen et al., 2020; Desimone 
& Pak, 2017; Kraft et al., 2018; Matsumura et al., 2010; Neuman & Cunningham, 2009), and the 
role of leadership in school reform (Desimone et al., 2019; Porter et al., 2000; Supovitz, 2015) 
demonstrates that LDC’s core components can be drivers of change in both teacher practice 
and student learning: 


e coach-supported professional learning communities formed at each school site to 
plan and support implementation of the LDC intervention and to provide a space for 
teacher learning and collaboration; 


e asynchronous support from coaches, primarily in the form of review and feedback 
through the CoreTools platform; 


e teacher implementation activities, including the creation/adaptation and classroom 
implementation of two LDC modules; and 


e leadership support at different levels with school and teacher leaders engaged in a 
process of goal setting, observation, and reflection related to the LDC work. 


LDC’s CoreTools online platform, with its module-building tools and extensive library of 
exemplars, tools, templates, and resources, provides the technological base for teachers’ work. 
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FIGURE 2 - LDC i3 Logic Model 
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District instructional specialists visit schools three times per year to sit in 
on PLC, meet with principal and teacher leader, and observe instruction. 
Principals participate in LDC by protecting PLC time, setting goals for 
LDC work, facilitating and participating in school visits from district 
instructional specialists, and observing LDC instruction. 
Teacher leaders facilitate PLC on the ground, collaborate with 
administrator, and communicate regularly with coach. 


Contextual Factors: District support, school and student demographics, other policies and initiatives 


The logic model predicts that these four components will lead to increased teacher pedagogical 
expertise and more effective CCRS-aligned instruction that incorporates ongoing formative 
assessment. In turn, this increased teacher capacity and more effective instruction will lead to 
higher student engagement in the short term; and improvements in student skill acquisition and 
test scores, and higher rates of course completion in the medium term, after 2 years of 
implementation. In the long term, LDC will improve students’ college and career readiness, 
education attainment, graduation rates, and labor market outcomes. The logic model provided 
the conceptual foundation for the evaluation study. 
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Evaluation Methodology 


The evaluation focused on teachers and their students in 37 district schools serving elementary 
and/or middle school grades, who participated in the study for at least 2 years. Schools and 
selected teachers within them entered the study in two cohorts; 20 Cohort 1 schools initially 
joined at the beginning of the 2016—2017 school year, and 31 Cohort 2 schools joined in 2017— 
2018. As of the final year of the study, 2018-2019, 11 of the original Cohort 1 schools were 
continuing to participate (8 Cohort 1 schools were not participating in LDC any more), as well as 


23 of the original Cohort 2 schools. 


Study data included a range of implementation measures, including teacher and leadership 
surveys, CoreTools processes data, analysis of teacher-created LDC units, and archival 
information on attendance and selected operations of LDC professional learning communities. 


LDC’s impact on student learning was gauged by a rigorous, quasi-experimental design using 
sophisticated statistical methodology. The analyses compared LDC students’ performance on 
Smarter Balanced, the state accountability test, in English language arts to that of a specially 
selected control group of students who were similar at baseline to the LDC group in 

demographic characteristics and prior achievement at both the school and individual student 


level. The study design meets the What Works Clearinghouse standards for rigorous research. 
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Study Findings 


In the following sections, we summarize study results organized by three categories of 
evaluation questions: What were the characteristics of the LDC sample? How well was LDC 
implemented? What were the effects of LDC on student learning? Unless otherwise indicated, 
the results are based on data from the study’s final year, 2018-2019, and combine results from 
both study cohorts. 


What Were the Characteristics of the LDC Sample? 


As Table 1 shows, the final study sample included 11 Cohort 1 schools, which began 
implementation during 2016-2017, and 23 Cohort 2 schools, which started at the beginning of 
the 2017-2018 school year. In Table 1, we provide a snapshot of school participation in 
2018-2019, the final implementation year. In 2018-2019, the majority of participants were from 
elementary schools and Cohort 2 schools. 


TABLE 1 - Number of Schools and Participants in 2018-2019 by Cohort and School Level 


Elem K-8 Middle 6-12 High a Keye-)| 
Cohort 1 Schools 7 O S O 1 11 
Cohort 2 Schools 16 2 4 1 O 23 
Cohort 1 Participants 48 O 23 O 9 80 
Cohort 2 Participants 155 21 35 21 O 232 
The data in Table 2 show that more than three quarters of the study students were of Hispanic 


origin and nearly 90% were characterized as “in poverty,” based on eligibility for free or 
reduced-price lunch. Black students comprised 10% of the sample, and Asian and White 
students constituted a lesser percentage, with 17% classified as limited English proficient and 11% 
and 14% categorized as special education and gifted respectively. Our analysis revealed these 
school and student characteristics were similar to those for the district as a whole. However, 
students in LDC schools also were slightly more likely to be living in poverty, and were slightly 


lower performing than all those in district schools serving students in Grades 3-8. 
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TABLE 2 - Characteristics of 2018—2019 LDC Schools and Grades 3-8 Student Population 
at LDC Schools Compared to the District as a Whole 


WY£-Tat=] 0) (3 LDC Sample Whole District 
Number of Students 15,953 23/,/28 
Student Level Characteristics 

Hispanic (%) 77 72 
Black (%) 10 8 
Asian/Pacific Islander (%) 4 6 
White (%) 7 11 
Female (%) 49 49 
Limited English proficient (%) 17 18 
Special education (%) "1 12 
Gifted (%) 14 15 
Free or reduced-price lunch (%) 87 84 


Student Achievement ? 


Mean standardized ELA achievement -0.08 0.00 


Mean standardized math achievement -0.06 0.00 


* Due to missing data, the number of students with achievement data is slightly smaller. For LDC, the ELA was 14,587 
students and the math sample was 14,696 students. Correspondingly, the district ELA and math samples were 219,795 and 
221,550 students respectively. 
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How Well Was LDC Implemented? 


Our evaluation foci mirrored the LDC logic model, as shown in F 


igure 2 above. Overall, the 


evidence suggests that implementation was generally faithful to the intended structure of LDC, 
although the level of implementation was not uniformly high. Nonetheless across the 3 study 
years, teachers and administrators consistently reported positive attitudes toward LDC and its 


implementation at their schools. 


PROFESSIONAL LEARNING COMMUNITY (PLC) COLLABORATION 


AND IMPLEMENTATION SUPPORT 


The frequency of PLC meetings varied greatly across schools, with some PLCs 
struggling to meet frequently and ensure high attendance, but a majority were able to 
achieve the “every two weeks” goal and found their collaboration with peers helpful. 


Teachers were nearly uniform in reporting that the 


r PLC teacher leaders were 


supportive, knowledgeable, and helpful. Teacher leaders similarly reported high 


ay 


satisfaction with Support from coaches, professional development offerings, and how 
the teacher leader role allowed them to be instructional leaders in their schools. 


TEACHER CLASSROOM LDC IMPLEMENTATION 


creation of new ones. Most teachers also reported implemen 
across the school year, as LDC expected. The degree of tea 
however, varied substantially, as did module quality, based o 
and our module analysis. 


Program data indicated that nearly all teachers participated in module building, and for 
many this involved collaboration and adaptation of existing LDC modules, rather than 


ting multiple modules 
cher engagement, 
n data from CoreTools 
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SUPPORT FROM COACHES 


Teachers nearly uniformly praised their LDC coaches for the training and 
implementation guidance they provided and found their coaches’ feedback useful 
to their teaching practices. However, our CoreTools analyses revealed that coaches 
did not use CoreTools as much as expected to provide feedback to teachers. 
Coaches met LDC minimum feedback expectations on 60% of the modules by 


providing at least two comments on each teacher module. 


LEADERSHIP SUPPORT 


Teachers also were almost universally positive about the support, encouragement, 
and feedback they received from their teacher leaders. Schoo! administrators were 
seen as generally supportive of the program, but their level of participation varied 
greatly across the sample. While school attrition rates speak to uneven leadership 
support, most schools that remained in the program at the end of the study 
indicated that they would continue to use LDC practices and tools. School 
personnel generally perceived district leaders as supportive of LDC, but a number 
id not feel that district leaders fully understood the program. 


(OL 


LDC IMPACT ON TEACHER SKILL AND STANDARDS-BASED CURRICULUM 


Teachers and school administrators uniformly felt that LDC had improved 
participants’ instructional planning and pedagogical skills, particularly in the areas of 
focusing on and creating standards-based assignments. The great majority also 
reported that LDC had helped teachers strengthen their formative assessment 
practice by helping them collect, analyze, and use evidence of student progress to 
modify subsequent instruction. Respondents’ views, furthermore, grew even more 
positive over time. Teachers and administrators also felt that LDC was improving 
student learning across multiple domains, in particular writing skills. 
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What Were the Effects of LDC on Student Learning? 


4+ 


Our quasi-experimental design analyses of student performance on Smarter Balanced 

confirmed both teachers’ and administrators’ positive views and LDC’s hypotheses that with 2 
years of experience, teachers would produce impact on student learning. All study analyses 
favored LDC students compared to their carefully constructed control groups, and effects for 
middle school students’ scores were statistically significant. Cohort 2 middle school teachers 
and students, who benefited from a refined LDC training approach, showed particularly 


strong effects. 


Our analyses employed two different approaches. The first model we termed “dosage 
dependent” because it took account of the reality that students, particularly at the middle school 
level, could vary in the number of LDC teachers they experienced—for example, they could 
have LDC teachers in English, social studies, and/or science. This model tests the extent to 
which students benefit from more exposure to LDC classes and teachers. The second approach 
we termed dosage independent because it considered all LDC students the same regardless of 
the number of LDC teachers they had. Study students were simply coded as exposed to LDC 

or not. 


TABLE 3 — Impact of LDC on State English Language Arts Assessment Scores by School Level 


LDC Analytical Sample Information 


School/teacher Level Samples | DYos-FaYe(-Mrel=Jel-lale(-sals 
with Two Years of LDC Schools Teachers Students Model Coefficient (SD) 
Cohorts 1 and 2 Elementary 17 34 685 0.028 (0.078) 
Cohorts 1 and 2 Middle 1 54 3,294 0.281 (0.081)* 
Cohorts 1 and 2 Elem and Middle 26 88 3,979 0.124 (0.054)* 

Note. Full results can be found in our earlier reports: Wang et al., 2019; Wang et al., 2020. *p:<205 


For middle school students exposed to LDC instruction in ELA, social studies/history, and 
science, effect sizes translated to a striking 9.4 months of additional learning compared to similar 
peers. The effect size for the average observed student (who received a smaller dose of LDC 
instruction) translated to a still very impressive 4.1 months of additional learning compared to 
similar peers (See the green box plot in Figure 3). 
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Our analyses also examined the effects of common covariates on the performance of students 
whose teachers had 2 years of LDC experience. The effects of the covariates were similar 
under both models and were in the expected directions. Baseline ELA performance remained 
the strongest predictor and baseline mathematics performance also helped explain student 
outcomes. In addition to baseline achievement, Hispanic students, students receiving free or 
educed-price lunch, and English language learner status were all significant predictors of ELA 
performance and were in the expected directions. Students enrolled in honors English courses 
performed at higher levels than did their peers taking standard English courses, and female 
students performed significantly higher. Students performed similarly in Cohort 1 and 2 schools. 


inally, Figure 3 depicts the estimated impact of LDC for students exposed to LDC teachers in all 
three major content areas: ELA, social studies/history, and science. Results are shown by school 
evel and combined and again highlight LDC effects at the middle school level. These effect 
sizes can be best understood as the estimated impact of LDC under ideal conditions. 


FIGURE 3 - Treatment Effect on Smarter Balanced English Language Arts Scores 
with 95% Credibility Interval for Students with Average LDC Dosage, by Cohort 


Teacher Cohorts 

E= Elementary Teachers (n=36) 

HB Middle Teachers (n=54) 

S Elementary and Middle Teachers (n=90) 
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Summary and Conclusions 


n summary, our multiyear, mixed-methods evaluation of LDC implementation in a large, West 
Coast urban school district provides evidence that LDC instruction improves student learning, 
particularly at the middle school level where results were statistically significant. Students taught 
by teachers with 2 consecutive years of LDC participation scored at significantly higher levels on 
their state accountability test in English language arts than the comparison group of carefully 
matched students in similar schools in the same district. The observed effect translates to 9.4 
months of additional learning for students exposed to LDC in all their core content areas 

English, history, science), and 4.1 months of additional learning for the average observed student 
in the study. The dosage-dependent model, in short, demonstrates well that middle school 
tudents with higher levels of exposure to LDC (by being taught by teachers in multiple content 
eas implementing LDC) benefited more from the program. It is also worth underscoring that 


a 
LDC achieved these significant effects in a low-performing urban setting, where students of 
C 
a 


S 


olor and students living in poverty comprised the great majority of the study’s student sample, 
nd a moderate proportion were not fully English proficient. 


Teachers were nearly uniform in their positive attitudes about the value of their LDC 
participation. Educators found LDC to be a helpful tool for fostering collaboration, creating a safe 
space for sharing practice, and strengthening their instructional design and pedagogical skills in 
literacy development. Teachers reported that LDC had improved their standards-based practice 
and their students’ learning, particularly in writing. 


Despite broad support and demonstrated impact, however, our results indicate that many 
schools and teachers did not fully meet LDC fidelity thresholds. They did not fully participate in 
professional learning communities with the frequency and duration that LDC anticipated and did 
not fully meet expectations for module development and implementation. Moreover, a number 
of principals or other school leaders did not meet LDC expectations for their involvement, which 
was emblematic of inconsistent leadership support in some schools, for example in the critical 
area of common planning time. Survey results, in fact, identified school and district leadership 
support as a relative weakness in LDC implementation. The responses of sizable proportions of 
teachers and teacher leaders, for example, indicated that principals did not observe LDC 
teachers as expected and did not offer feedback or participate in LDC planning and 
implementation. Principal support, in these instances, thus was more hands-off than hands-on. 
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Yet strong leadership support is essential for the implementation of new programs and practices 
Desimone et al., 2019; Porter et al., 2000; Supovitz, 2015), and it seems axiomatic that 
incomplete implementation hampers program impact. Study findings, in short, highlight the 
importance of school leaders and teachers understanding and committing to the program in 

ion, and the need for school leaders dedicating substantial attention and 


advance of implementat 
esources to the program. 


That LDC demonstrated stronger effects at the middle school level also is noteworthy, mirroring 
research showing differences in the uptake of reforms for different groups of teachers (Edgerton 
& Desimone, 2018). In exploring these differences, LDC may wish to investigate strategies to 
increase its elementary school impact. Study findings on the effect of LDC dosage suggest that 
one key strategy lies in increasing elementary school students’ engagement with LDC modules. 
That is, for typical elementary school classrooms taught by one teacher, study students had the 
opportunity to engage at most with two LDC modules over the course of the year, but at the 
middle school level, with different teachers for English, history/social studies, and science, 
students’ potential engagement was six modules. Perhaps grade-level professional learning 
communities in elementary schools could be charged with developing or selecting larger pools 
of curriculum-appropriate modules in various Subject areas for implementation in all grade-level 
classrooms and multiple subjects, thereby increasing students’ exposure to LDC-infused 
instruction. 


Study findings also underscore the value of all students’ repeated immersion in LDC. LDC and 
implementing schools may want to consider how they can maximize students’ LDC dosage, for 
example through cross-disciplinary grade-level teams and cross-grade disciplinary teams to 
embed LDC more fully within and across grades. Under these optimal conditions, LDC may 
show even greater impacts on teacher practice and student learning. 
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